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Abstract. There is a commutative algebra of differential-difference operators, 
acting on polynomials on R 2 , associated with the reflection group B2. This 
paper presents an integral transform which intertwines this algebra, allowing 
one free parameter, with the algebra of partial derivatives. The method of 
proof depends on properties of a certain class of balanced terminating hyper- 
geometric series of 4i*3-type. These properties are in the form of recurrence 
and contiguity relations and are proved herein. 



1. Introduction 

1.1. Overview. We construct an integral for the intertwining operator V as- 
sociated to the reflection group of type B2 (order 8) acting on R 2 , with one param- 
eter k. For polynomials or adequately smooth functions in x — (2:1,2:2) define the 
differential-difference operators: 

(1.1) T lf (x) = — / (x) + m 

f (x) -f(x 2 ,xi) , / (x) - f (-x 2 , -xt) 

Xl - X 2 Xl + X 2 
T2f iX) = dx-J {X) + Kl X 2 

f(x) - f(x 2 ,xi) f(x) - f(-x 2) -xi) 

+ K2 \- K-2 • 

X 2 - Xl Xi + X 2 

These operators are special cases of those defined by the author in [2]. Their key 
property is commutativity, T1T2 = T 2 T\. We deal only with the restricted case 
Ki = K2 = k. The intertwining operator V preserves the degree of homogeneous 
polynomials and satisfies V ^g§-^ f ( x ) — TiV f (x) for i = 1,2, and VI = 1. In 

[3] it was shown that V exists and is one-to-one as a map on polynomials for any 
k except for the set { — ^ :mGN,^ ^Z} of singular values. The definition and 
existence of V for n > was shown in |3]. Later Rosier |8] (see also |9]) proved 
that the functional / 1— > Vf (x) is given by integration with respect to a positive 
measure, for each x £ R. An explicit integral was found for the group S3 (symmetric 
group on 3 objects) in |4]. Xu p.0 found an intertwining transform for B2 under 
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restrictive conditions on degree and «i,«2. We will use a criterion, valid for any 
finite reflection group, for an operator to be equal to V which was proven in that 
paper. There is a kernel which contains all the information about the action of V 
on polynomials. For ijEl 2 let (x, y) := J2i=i x iVii then 

K{x,y) = V x (exp (x,y)) 

is entire in x and y (V x acts on the variable x). Further let K n (x, y) — -^jV x ({x, y) n ) 
for n G No := {0,1,2,...}. The book of Dunkl and Xu |5] is a reference for 
differential-difference operators, the intertwining operator and the kernel K (x, y) 
for any finite reflection group. Asymptotic formulae for the kernel are discussed in 
[5] . The starting point for the S3 result was an integral formula of Harish-Chandra 
involving integration over a compact Lie group. This gives a special case (n = 1 
and symmetrized) of the intertwining operator for the associated Weyl group. 

1.2. The Symplectic Group Integral. We use an approach similar to that 
in |1| sect. 2]. Consider B 2 as the Weyl group of the (compact) symplectic group 
Sp (2). This group can be described as the group of 2 x 2 unitary block matrices 

U =(-B *)' UU " = '- 

where A, B are 2x2 complex matrices. The subgroup 

T= {diag(e iei ,e ie2 ,e- i9l ,e- ie2 ) :9 1 ,6 2 eR} 

is a maximal torus ("diag" denotes the 4x4 diagonal matrix with the specified 
entries) . Identify M. 2 with the complexification of the Lie algebra of T by the map 
5 : x 1— > 2~ 1 / 2 diag [x\, x 2 , — X\, — x 2 ) (the purpose of the factor 2" 1 / 2 is to get 
Tr(5(x)5(y)) = (x,y)). The formula of Harish-Chandra (Helgason p.328]) 
specializes to 

exp{Tr{U5(x)U 8{y)))dm(U) = 2 - , 

s P (2) 2 p{x)P(y) 

where dm (U) is normalized Haar measure on Sp (2) and p (x) — x\x 2 {x\ — x^). 
The right side of the formula is an expression for the kernel K° (x,y) at k — 1. 
Thus the left side suggests a construction of an integral formula for the intertwining 
operator. We compute 



Tr(US(x) U*S(y)) (|A ri | 2 - \B^ 2 

i=i j=i 



Thus one needs to integrate functions of the four variables ^|A,i| 2 — li^ 2 ^ with 
respect to Haar measure. One applies integration over the subgroup Sp (1) and the 
homogeneous space Sp (2) /Sp (1). Since this only gives the n = 1 situation more ex- 
perimentation is needed to make a conjecture about arbitrary k. By direct polyno- 
mial calculation we find the symmetrized kernel K® (x, y) — \ J2w<eb 2 V x ( xw > v)™ / n - 
for small n (< 8) and try powers of the trigonometric functions to produce K® (x, y) = 
^7 J n (xt (q) , y) n dfj, (q). This approach, however, has as yet not produced a solu- 
tion for the two-parameter situation. 
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1.3. Group actions and the measure. The group B 2 is generated by the 
-1 \ / 1 

1 and CT2 := 1 



reflections <ti := I ^ !?) and cr 2 := ( !? 1. I . The intertwining operator has 



the form 



Vf{x)= I f(xT(q))g{q)dp(q). 
r(q) = 



n 

91 93 

92 94 

where /x (g) is two-sided invariant for B 2 and 17 (g) is invariant under the action 
w : t (q) — > wr(q)w^ 1 for all u> 6 £? 2 (actually g(g) involves one more variable 
of integration). There are left and right representations of B2 on q (that is, R 4 ), 
defined implicitly for w G B2 by 

r(q\(w)) = w~ 1 T(q) , 
T{qp{w)) =r(q)w. 

For example qX(ai) = (-91,92,-93,94) and qp(a 2 ) = ((73,94,91,92)- The invari- 
ance conditions are dp (qX (w)) = dp (qp (w)) = dp (q) and g (qX (w) p (w)) = g (q) 
for all w £ B 2 . It suffices to check invariance for the generating reflections, that 
is, g and p must be invariant under q 1— ► qX (01 ) p (01 ) = (91,-92,-93,94) and 
q 1 ► gA (02) P (02) = (94,93,92,91), and additionally /x must be invariant under 
9 ^ 9A(<Ti) = (-91,92,-93,94) and <? gA(<7 2 ) = (92,91,94,93)- 
The measure /x is actually an integral over six variables: let 

qi = u cosf/'i 

92 = (1 - u) cos ^2 

<73 = (1 — u) (cos V>2 cos 9 + sin ^2 sin 9 cos 02 ) 
(74 = u (cos V>i cos 9 + smtpi sin cos 0i ) ; 

the region f2 of integration is < u < 1, < 6, (pi, ipi < tt (i = 1, 2). For k > i the 
measure is 

d/x (q) = c K (u (1 — u) sin^i sin ^2 sintf) 2 "' -1 (sin 4>i sin</> 2 ) 2K ~ 2 dudi\)\di\) 2 d9d<p\d<p 2 

times the normalizing constant (product of gamma functions) so J n dp = 1 . We 
can now state the main result for homogeneous polynomials: let 

°° = ^ + ^ {dq-^Wj-^-^ Wi-dqS 
and suppose / (x) is a homogeneous polynomial in x of degree n. 

(1) If n is odd, then 

(1.2) Vf (x) = [ 2 ( gi + g 4 ) / (st («)) d/x (g) . 

Jo 

(2) If n is even, then 

(1.3) Vf (x)= J jl + (j! ,74 - 9293 + ^^A) j / (r (9) x) dp (q) . 
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Further, the Bessel function K°(x,y) — | J2weB 2 ^ ( xw > V) IS gi yen by the 
positive integral: 



K°(x,y)= / exp((xT(q) ,y))dfi(q) . 
Jn 



We will consider the relation to the representations of B 2 and the techniques of 
proving the formulae in the following sections. The required invariance properties 
of \i are made clear by a change-of-variables. 

Lemma 1. In terms of q,6,u and the auxiliary variables 

u 2 sin 2 9 — qf — q\ + 2qiq 4 cos 9 



Zi {qi,q4,u,6) 



sin 



2 , 



(1 - M) 2 sin 2 6> - q\ - q\ + 2q 2 q 3 cos9 

Z2 (92, 93, u, 9) = r-g- 

sin 9 

the measure is given by 

d/i = c K (ziz 2 ) K 3//2 (sin#) 2K 3 dqidq 2 dq 3 dq^dud9 ', 
and i/ie region of integration Q,\ is implicitly defined by z\ > 0, z 2 > 0, < 9 < 

7T,0 < U < 1. 

Proof. In terms of qi,q 2 , 93, 94, u, the Jacobian is 

r d(q 1 ,q 2 ,q 3 ,q 4 ,u,6) . . . 2 . . 

J = — — ■ ■ ■ — = (u (1 - u) sin -01 sm?/>2 sintM sm0i sm0 2 . 

d (0i, 2 , V'l, ^2, 

Then 

<i/i = c K (u (1 — u) sin^i sin ^2 sin0i sin0 2 sin#) 2K ~ 3 Jdudipidip 2 d9d(j)id(j> 2 . 
Observe u sin tpi sin cos 0i = 94 — 91 cos and u 2 sin 2 0>i = m 2 — g 2 thus 

. 2 f 94 - <7lCOS^ X 

sin 0i = 1 ' 



u sin "01 sin# 



/ • , • , n2 / 2 2\ (94 -91 COS 0)' 

(ti sin ^1 sin 0i ) = (u -q 1 ) —5- 



sin 

= Z-i (qi,q 4 ,u,9) , 



and similarly 



((1 - u) sin V2sin0 2 ) 2 = ((1 - u) 2 - qfj 



(93-92 cos 9) 



2 



sin 2 9 
= z 2 (q 2 ,q 3 ,u,9) . 

Thus dfi = c K (2iZ 2 ) K_3 ^ 2 (sin6*) 2K_3 dq 1 dq 2 dq 3 dq^dud9 . □ 

The measure \x is invariant under the transpositions (91, 94) , (92, 93), and the in- 
volutions q i—> q\ (01) = (172, 91, 94, 93) and 9 h-> 9A (ai) = (—91, -92, 93, 94) because 
of the equations 

21 (92, 93, 0) z 2 (91, 94, u, 0) = zi (91, 94, 1 - u, 9) z 2 (92, 93, 1 - u, 9) , 
Zi (~qi,q4,u,9) z 2 (-q 2 ,q 3 ,u,9) = Z\ (qi,q 4 ,u,ir - 9) z 2 (q 2 , 93, u, ir - 9) . 

Similarly there is invariance under the right action of B 2 , that is, 9 1— ► qp (at) ,i — 
1,2. 
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To integrate the typical monomial q a := q^q^ 2 <h 3 1i 4 where a £ Ng (q a is of 
degree \a\ := X)i=i a ») set 

foo = (012 + a 3 ) /2, 
61 = (ai + 04) /2, 

fe 2 = («2 + Oti) /2, 
&3 = («3 + 0:4) /2, 

and let s (a\, 0*2, as, a±) := J n q a d[i (q). By a parity argument the nonzero integrals 
occur only for integer values of the hi. 

Proposition 1. The normalizing constant is 

(2«-i) 2 r(2« + i) 

and for a £ Nq if ai = a 2 = 03 = «4mod2 faZZ even or all odd) then 

/■t a \ / \ ( 2k ) 2 &i ( 2k )26o (Dfci (2)60 (2)63 

(L4a) S(ai ' a2 ' a3 ' a4H (4«W 26o («^) 6l (« + |) 6o («^) 63 

x ^ L "f 2J (~"4) M (-^) 2J 2 _ 2t _ 27 

otherwise s (a\, a.2, 0:3, 0:4) = 0. 

PROOF. Expand in terms of u,0,i/ji,ip2,<pi,<p2 with the binomial theorem 
and collect terms. The result is 



S (o„<«,ob,o 4 ) = egg jf (1 - „)«+».+"•-' du 

Jo 



x / (cosi/r)" 1 ^" 4 ' (sin^i) 1-1- ^ / (cos^i)' (sin^i)' i ' i 2 d(j) 2 



f 

Jo 

f 

Jo 



\ai+Q4 


1 (sin?/>i) 4+2K 1 dip 2 


/ (cos <J 
'0 


!>i) 4 (sin0i) 2K 


2) 


~* (smip 2 y +2K ~ 1 dipi 


r 

/ (cos ( 


fe)- 7 (sin0 2 ) 2K " 


1 0:3+0:4-1 


l ' 3 (smey +J+2K - 1 de. 







x / (cos6>) 



Recall Jq cos™ sin A 9d0 equals zero if n is odd and equals B ^2^) if " is even, 

for n = 0, 1, 2, . . .and A > — 1. For the respective integrals to be nonzero i and j 
must be even because of the 4>\ and 4> 2 integrals and hence a 2 + 03 , a± + a 4, 0:3 + 
must be even (in the ipi,ip 2 ,8 integrals). So replace i,j by 2i,2j respectively and 
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assume that the entries of a are all even or all odd. Then 

Lq 3 /2J L«4/2J 



s(a 1 ,a 2 ,a 3 ,a 4 ) = c K ^ ( 3 J ( .J B (ax + a 4 + 2k, a 2 + "3 + 2k) 

j=0 i=0 

/ai+a 4 + l . . \ / 1 1 
x ±J | - i,z + k I i? U+ 2' K — 2 



Ct\ - 


h CK4 - 


h 1 




2 




Q!2 " 


h «3 - 


r 1 




2 




a 3 - 




- 1 


2 



x B ( ^ + k 

We set a = to find the normalizing constant, indeed 



f \ 3 „ fl I x 2 



The stated value follows from the duplication formula. The following ratios are 
typical in the calculation: 

B{i + \,K-\) r(< + !)r(«-!)r(«) _ (£). 



b{\, k - i) ~ r( K + i)r(|)r( K -i) ~ («),' 
g (ftp + 1 - i, i + «) = r (bp + 1 - i) r (t + «) r (« + 1) = W t (^) bo _, 



Thus 



L-W2J L«W2J , x , x (2k) + (2k) + 
(aifeo-j (2)^ \2)b!-j (a)6 3 -i-j 



63 



To finish the proof, write (£) = (-a 4 ) 2l / (2 2 ^! (§).), = (-1)* (±) bo / (§ - &o), 

and similarly for the other terms. □ 

It is clear that the symmetry s (ax, a 2 , a 3 , a.4) — s (a 2 , cti, ax, a 3 ) holds (as 
well as s (ax, a 2 , a 3 , ax) = s (ax, a 3 ,a 2 , ax)), as implied by Lemma ^ Some other 
symmetries will be shown later. Singular values are numerical values of k for 
which the intertwining operator does not exist. By the general theory of singular 
values specialized to B 2 they consist of -| - N U (-| - N U -f - No). The 
denominators (k + g), in s (ax, a 2 , a 3 , ax) correspond to the first subset. For the 
second subset consider 

(2«) 26l (2k) 

2fco _ 2-2bi-26 _ 

(2 K ) 2bi (2k) 

2b 

( 4K )2 fcl+26o (2K) bi+bo (2K+l) bi+bo 

= 2 -2 bl -2b ( 2K + fc l + b 0) bl -b„ ( 2 ^2b 
( 2K+l 2) bl+b0 

if bx > 60 or a similar expression if bo > b\. 
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To see some of the complexity of this integral: consider the reduced form 

/ (9i,92) dfj,(q) = 



J 



f (ucos tpi, (1 — u) cos ^2) {u (1 — u) sinV'i sin?/> 2 ) 2K 1 dudipidtp2, 

10 Jo Jo 

with a constant c depending on n. This is not reducible to an ordinary double 
integral without the use of a nonelementary integral. Indeed, 



f(Qi,Q2)dfj,(q)= / f(q 1 ,q 2 )E K (qi,q 2 )dq 1 dq 2 , 

Jn J J|?i|+|g 2 |<l 

E K ( qi ,q2) = cJ [ (u 2 - q\) [(1-u) -ql)) du. 



191 1 

1.4. Single sum formula and hypergeometric functions. Even though 
our evaluation of s (ai, a 2 , «3, 0:4) required a six- variable integral the value can be 
expressed as a single sum in terms of a terminating balanced 4F3 hypergeometric 
series. 

Write 

, x ( 2k )26i ( 2K )2fc / 1 \ 

s {ai,a 2 ,a>3,a4) = , . s (ai, 02, 0:3, cu) , 

\ 4:K )2b 1 +2bo 

then s' satisfies the recurrence 

(1.5) aia4 + ^ (a 2 + «3 + 1)^ s' (ai - 1, a 2 + l,a 3 + 1, a 4 - 1) 

+ 2 ( a 2«3 ("i + Q!4 + 1) - ai« 4 (o!2 + a 3 + 1)) s' (ai, a 2 , a 3 , 04) 

= a 2 a 3 ^ + ^ ( a i + a 4 + 1)^ s' (ai + 1, a2 - 1, «3 - 1, 04 + 1) . 

There is a good reason why this formula appears here. The intertwining operator 
will be described as a linear functional on the space of polynomials in q applied to 
polynomials in it (q) = (x\qi + x 2 q 2 ,x±q3 + x 2 q^). The coefficient of x\ +b ~ c x% in 
the expansion of {x\q\ + x 2 q 2 ) a (x\q3 + x 2 q4) b is the following: 

min(6,c) . . , . 

(!-6) Pa, b (l)= E c !j ( i )£- C+i &- i <t i Qi 

The need to integrate these polynomials motivated the examination of and ex- 
perimentation with s (a — c + i, c — i, b — i, i) as a function of i. This led to the 
discovery of the recurrence which in turn suggested that there might be a single- 
sum form of s. It turns out that the proof of the recurrence actually uses the single 
sum, a terminating hypergeometric series of ^F^-type (the argument looks circular, 
but the proof of the single sum does not use the recurrence). 



Proposition 2. Suppose a 4 = and a. t = 2$ for 1 < i < 3, then 
S '(2/3 1 ,2/3 2 ,2/3 3 ,0) 



(2)/?! (2)^2 (2)^3 



= s' (0,2/32,2/33,2/?!), 
and the value is symmetric in (5\ , {3 2 , . 



s 
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PROOF. Indeed the sum in s' (2/3i, 2/3 2 , 2/3 3 , 0) equals 
(|-/3 2 -/3 3 -«)„ ( K + /3 2 + i) 



( K + ^fe+fe (life 

(by the Chu-Vandermonde sum) and this proves the first equation. The second 
equation follows from the symmetry of Lemma ^ The sum in s' (0, 2/3 2 , 2/3 3 , 2/3i) 
can also be found directly, first summing over < i < fl\ and using similar argu- 
ments as in the first equation. □ 

Theorem 1. Suppose a 6 Ng and a\ = a 2 = 0.3 = ct\ mod 2. Lei 6; = 
(cXi + en) /2, 1 < i < 3, i/ien 

/I -7\ / / \ (2)^ ( 2 )b 2 (2)63 

(1.7) s (ai, a 2 ,a 3 ,a 4 ) - 1 - 



(«+l) 6l («+3) b2 («+3) fc3 

La ^ 2J (-f),( 1 f),W,H-^M, 

T/iis is a terminating balanced 4F3 series. 

The proof of this key result is in Section 3. To say that the series is balanced 
means that the sum of the denominator parameters equals one plus the sum of the 
numerator parameters; the property is also called Saalschutzian. The particular 
choice of parameters for this 4-F3 series will appear often in the sequel and so we 
make the following: 

Definition 1. For n G No and free parameters u, v±, u 2 , v 3 let 

, „ ( -f , ^,u, ~u - Vi - v 2 - v 3 \ 
F {n;u,vi,v 2 ,V3) = 4F3 , , , ;1 . 

^ 5 - vi - n, J - v 2 , 2 - v 3 J 

This is a balanced terminating hypergeometric function. 

This is not the generic balanced 4_F 3 -series (which has 6 free parameters) be- 
cause of the parameters (— ■ With this notation the Theorem can be restated 
as: 

Theorem 2. Forcti = a 2 = a 3 = 014 mod 2 the following single-sum expression 
is valid: 

s (q.) — (^ K )ai+Q4 i^ K )a 2 +a 3 (2) 6i (2) b 2 (2) b 3 



( 4k )| Q | ( K +5) bl ( K +^) b2 ( K +5) t 
x F j a 4 ; «, ^ (ai - a 4 ) , ^ (a 2 + a 4 ) , - (a 3 + a 4 ) 
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1.5. Contents. In Section 2 we describe the representation-theoretic impli- 
cations of the invariance conditions, present an overview of the method of proving 
that a given linear functional on polynomials in q produces the intertwining op- 
erator and then give the actual proof. Some ingredients of the proofs depend on 
contiguity relations for the function F. Also there is a purely integral form of V 
not involving the degree of polynomials. 

Section 3 contains the proof of the single-sum result, which relies on classical 
transformations of hypergeometric series, and the (computer-assisted) proofs of the 
required contiguity relations of F. There are closing comments in Section 4. 

2. The Intertwining Operator 

2.1. Invariants and representations of B 2 . The irreducible representa- 
tions of B 2 are realized in the space of polynomials 1R [£1,2:2] as follows: there is 
one of degree 2, which is realized in each space of polynomials homogeneous of 
degree 2n — 1 as n isomorphic copies 

{c x 2 1 n - 1 - j 4 + cia^ n_1 - j : < j < n - l} , 

for n = 1, 2, . . .; there are four one-dimensional representations: 

(1) invariants: x 2n 3 x 2 + x 2 n 2j , < j < n; 

(2) determinant: x 2n -^x^ 1 -x^ +1 xl n ~ 1 ~ 2i , < j < [f J -1 and 2n > 4; 

(3) type 1: x\ n x 2 2 i+1 + xl^xf' 1 '^ , < j < [^\ and 2n > 2; 

(4) type 2: x 2n ~ 2i x 2 j - x\ j x 2 * ~ 2j ,0 < j < [^\ and 2n > 2. 

The reason for naming types is that <ii acts as multiplication by — 1 on type #i 
for i = 1, 2. Consider polynomials in q invariant under q 1— > q\ (w) p (w) , w e B 2 . It 
is easy to compute the Poincare series for the ring of invariants (graded by degree) , 
namely (j—^yfz^z an d then describe the ring as 

(1 © (<?i - q A ) (ql - ql)) R [q x + <? 4 , q\ + qj, q\ + g|, q 2 q 3 ] . 

The invariant of degree 1 is associated with the two-dimensional representation of 
B 2 and we set 

go (q) = 2(qi+ q 4 ) . 

There are 4 linearly independent invariants of degree 2, the two-sided J2i=i ih an d 

91 (<?) = 9i94 + 9293, 

92 (9) = \ {ql - ql - ql + ql) , 

93 (9) = 9i94 - 9293- 

Then 

.91 (9^ (01)) = -.91 (9) ,52 (9A (01)) = .92 (9) ,.93 (9A (01)) = -.93 (q) , 

91 (9^ (<7 2 )) = 91 (9) , .92 (9A (cr 2 )) = -.92 (9) , .93 (9A (<T 2 )) = -fif 3 (9) • 

Further (1,0, 0,1) = 1 for 1 < i < 3 (a plausible normalization). By simple 
orthogonality arguments we see that if / (x) is a polynomial of even degree then 
J n f (xt (q)) gi (q) dfi (q) ^ only if / (x) has a nonzero component of determinant 
type for i = 3, or type i for i = 1,2. Experimentation quickly showed that the 
formula Vf (x) = j n f (xt (q)) 53 (q) d[i (q) appeared to be valid for the determinant 
type, but the similar attempt failed for types 1 and 2 (in fact, no polynomial in q 
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of degree less than twelve with the correct behavior under the 2?2-action works). 
We comment on this quandary in Section 4. We set 

d 

Si = «-,!<*< 4. 
oqi 

The formula for V comes from applying the adjoint action L := ad ( | J2i=i ®i) to 
(multiplication by) the gi (recall (ad (A) B) f := (AB — BA) f for operators A, B 
on polynomials /). Indeed 

L (5i) = 91^4 + <74<9i + q 2 d 3 + q 3 d 2 , 

L (g 2 ) = qidi - q 2 d 2 - q 3 d 3 + q 4 d 4l 

L {93) = 9i<9 4 + <?4<9i - q 2 d 3 - q 3 d 2 . 

These operators have the same invariance properties as the respective polynomials 
gi. Set D := L (g x + g 2 ) = {qi + q 4 ) {di + d 4 ) - {q 2 - q 3 ) {d 2 - d 3 ) and D 3 := 
L(g 3 ). 

The formula stated in the introduction can be given as a pure integral with no 
derivatives for n > |. However another variable of integration occurs (a total of 
seven!). Let u> denote the integration operator: 

wf{x) := ( /(to)* 4 "- 1 *, 
Jo 

thus ujf (x) = 4l ^ +n f (x) when / is homogeneous of degree n. 

Theorem 3. Let k > | and suppose f is sufficiently smooth on a ball Br — 
{1 £ I 2 : ||x|| 2 < R] for some R > 0, then for x 6 Br 



Vf(x)= / f(xr(q))(l + g (q)+g 3 (q))d f ,(q) 
Jn 

+ (2k-3) / ujf '(xT(q))g o (q,u,0,<f>i,<j)2,ipi,ifo)diJ,(q) . 
Jn 



go 



qi + qi \ i / 92-93 



l + cos0 y u sin (f)\ sin if)\ J 1 — cos# \ (1 — u) sin cf> 2 sin ^2 

If / (x) is homogeneous of degree n then J„ / (xt (q)) g 3 (q) d/j, (q) is zero when 
n is odd and equals Sf } + - D 3 f (xt (q)) d/i (q) when n is even (the proof is in the 
next subsection). Also we will prove the two equations (|1.2|l and The factor 

4 . n in can be replaced by the integral operator uj. The following uses an 

integration by parts to replace the differential operator Do by an integral, the last 
ingredient of the formula in the theorem. 

Lemma 2. For k > | and a smooth function h (q) 

D h{q)dn{q) = (2k - 3) x 

h i q \) 1 ( 91+94 \ 2 1 / 92-93 V 1 d (q) 

[ 1 + cos 6 \ u sin <\>\ sin ipi J 1 — cos 9 \ (1 — u) sin <f> 2 sin ij) 2 J J 

Proof. Use the notation and change-of- variable from Lemma \I\to set up the 
integration by parts. The measure is dfj, = c K (ziz 2 ) K 3 ^ 2 (sin 9) 2k 3 dq\dq 2 dq 3 dq 4 dud9 
and the region of integration fii is implicitly defined by z\ > 0, z 2 > 0, < 9 < 
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7r, < u < 1. If w (9) vanishes on the boundary of the bounded domain Q in R 4 
then J Q D Q h (q) w (q) dq = - J Q h (q) D w (q) dq, (although f Q qj-^-h (q) w (q) dq = 

— fqh (q) fl + Ij^f^j w (q) dq for 1 < j < 4 the terms h (q) w (q) dq cancel out). 
We find 

(1 -cos 6>) (gi +q 4 ) 2 Aqi+qif 
sin 2 6» l + cos6< 

(l + cos6»)(g 2 -<73) 2 (<72-93) 2 
sin 6 1 1 - cos 

and D (ziZ 2 ) K_3/2 = («-§) (^f" + ^f) (zi^f 372 - Then change back to the 
original variables <f>±, 02, ipi, ip2, u, 9 to complete the proof. □ 

2.2. Proof of the intertwining property. Suppose £ is a linear functional 
on polynomials in q = (91,92,93,94), and define an operator on polynomials in 
x = (£1,2:2) by Vi/(a;) = £f (xiq± +X292, 2^193 +^294)- What needs to be done to 
show V\ = V, the intertwining operator? The group invariance requires £p (9) = 
& (94, 93, 92, 9i) = & (9i, -92, -93,94)- 

Definition 2. Let £0 denote the functional defined by q a t— > s(a). .For a 
homogeneous polynomial p (9) m 9 0/ degree n define the functional £ fry: 

£ (p) = £0 (50P) , /or n odd, 

£ (p) = £0 ( ( 1 + — — — D + — ^ — D 3 ) p (9) ) , for n even. 
\\ 4k + n 8k + n J J 

Using 4, Prop. 1.3] we get the homogeneous component of the criterion: 
(2.1) (n + 1) (<<r, y) £ {{xr (9) , y) n ) - £ (<xr (9) , y}" +1 )) 

»=i 

for n = 0, 1, 2, . . ., where {at} is the set of reflections {01, o"2, 02O1O2, 01O2O1}, and 
o"jT (9) = t (9A (o"j)). Rewrite the criterion as: 

£ ((9! + a 4 ) (xr (9) , - (» + 1 + 4k) £ ((xr (9) , 2/) n+1 

4 

+KEe(<^(9A( ( T l )),2/)" +1 ) =0 

i=l 

Note (9i + ^4) (xt (9) , y) = (x, y). We will prove £ satisfies the criterion and also 
that £0 (£*3P (9)) = (8k + n) £0 (jjsp) when p is of degree n (both sides vanish when 
n is odd). We have 

9A (0-1) = (-91,92,-93,94), 

9A (o- 2 ) = (92,91,94,93), 
9A (0-10-20-1) = (-92, -91, -94, -93) , 
9A (0-2O1O2) = (91,-92,93,-94)- 
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Then 

4 

^2g j (qX(a i ))=Q,j = 0,1,2, 

i=l 
4 

X]^3 (gA(cTi)) = -4 53 (?). 

The corresponding differential operators L(gj) satisfy similar equations for j = 
1,2,3 because Ylt=i ®1 i s two-sided invariant, that is, Ylt=i L (dj) P (l^ ( CT «)) = 
for j = 1, 2 and L (g 3 )p (qX (<n)) = -4L ( ff3 )p (?)• 

The proof of the criterion is easy when n is odd. 

Proposition 3. Suppose n is odd and p(q) is homogeneous of degree n + 1, 
then 

Co (go(q) (ft+ft)p(?)) 
- + 1 + ^0 ((l + I^T^o + ^ T - 1 D 3 ) P (q)) 

+ ^(( 4 -8^TT D3 ) p(9) ) = °- 

Proof. In fact, the left side simplifies to 

£o ((2 (qi + q±) (ft + ft) - (n + 1) - £>„ - D 3 ) p(q)). 

Replace n + 1 by the Euler operator X^=i 1^ then the expression becomes iden- 
tically zero. □ 

Corollary 1. The criterion \2.1)) is satisfied for odd n. 

PROOF. Set p (q) = (xr (q) , y) n+1 in the Theorem. □ 
For the odd degree case (n even) replace n by 2n, then the criterion becomes 

(2.2) (4k + 2n + 1) Co ((xr (q) , y) 2n+1 g, (<?)) - £ ((ft + ft) (xr (?) , y) 2 " +1 ) = 0. 

This equation can be restated as 

(4k + 2n + 1) Kan+i v) = 2/) ^2n y) ■ 

Recall the definition of P£ b (q) from Ql.fcif) . Thus 

(^r (?) , „)* = e r br^ E (?) *r c *§, 

t=0 ^ ' c=0 

for n G No- So it suffices to prove identities involving £ and (ccr (?) , y) n for the poly- 
nomials P^b- There are two immediate consequences of the invariance properties 
of£. 

Pl b ( q X(a 1 ) P (a 1 )) = (-l) b+c Pl b (q), 
Z(Pl b )^0^b = cmod2, 
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and 

■ya+b—c 



With the aim of applying condition l|2.2|) to P£ b with a + b = 2n + 1 we can 

assume that a is odd, b and c are even. This implies £o (^Pa & (9)) = (the 

typical monomial is q1 c+l q 2 ~ l q 3 ~ l q 1 ^ 1 ', the parities of the exponents of 53 and 54 
are opposite). 

In the evaluation of £ {^JqP^ b ^j and £ H0i + $4) -P^ fc) there are several van- 
ishing terms: 

Co (<z 4 P a %) = Co (0 4 P a c , 6 ) - £0 MAP Q c ib ) - 0, 1 < i < 4, 



6 («40?i£*) - Co (9i9 4 9iP a c , h ) = Co (<? 2 3 0iP a c , fc ) = eo M 2 0iP a c 



Thus 



^ (A> (fc + 0«) P a c , fc ) = Co (i (52) 0iP a %) + Co (L ( ff i) 4 P a c , fc ) , 
6 (D 3 (0i + 4 ) P a c , fc ) = £0 (Dzd A Pl b ) . 



The required identity is 



(2.3) (4k + 2n + 1) £ (2<?iP a c , b ) - £0 (fl + (52) 



0iP a c ,, 



(2k + n) (4k + n) 
= 0. 



$0 ({(3k + n) (<j 4 0i + 9i 4 ) + k (q 3 d 2 + q 2 d 3 )} 4 Pa, h ) 



We will prove this by summing over the monomials in P^ b , that is, we replace P£ b 

in the left side by qi~ c+t q 2 ~ z q 3 ~ z q\ and evaluate £0 ( m terms of s (a)). The partial 
sums are found explicitly by use of contiguity relations for the 4P 3 -type function 
F. 

Set a = 2ai + 1 + 2a 3 , b = 2a 2 , c = 2a 3 and so n = ai + a 2 + a 3 and assume 
for now that a± > 0, that is, a > c. Evaluate the left side for the monomial q a 
with a = (2a% + 1 + i, 2a 3 — i, 2a 2 — i, i) (note |a| = 2n + 1). One of the terms 
simplifies: 

. / 2ai — 2a 2 — 2a 3 — 4i\ 

(2ai + 1 + i) 1 + — , „ s (2oi + i, 2a 3 - i, 2a 2 - i, i) 

\ 4k + 2n / 

= 2 (2ai + 1 + t) ^ + fll — — s (2ai + i, 2a 3 - i, 2a 2 
2k + n 



To remove some common factors we divide by s (2a± + 2, 2a 3 , 2a 2 , 0) (see Propo- 
sition!^ for the evaluation) and denote the result by ij. The following expression is 
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a linear combination of F values with simple coefficients 

s (2ai + 2, 2a 3 , 2a 2 , 0) U = 2 (4k + 2n + 1) s (2ai + 2 + i, 2a 3 - i, 2a 2 - i, i) 
(2ai + l+i) (K + ai+i) 



2k 

i (3k + n) 



-s (2ai + i, 2a3 — i, 2a 2 — i, i) 
■{(2a\ + 1 + i) s (2ai + i, 2a 3 — i, 2a 2 — i, i) 



(2k + n) (4k + n) 

+ (i - 1) s (2ai + 2 + i, 2a 3 - i, 2a 2 2)} 

-{(2a 3 - «) s (2ai + 1 + i, 2a 3 - i - 1, 2a 2 - i + 1, i - 1) 



(2k + n) (4k + n) 

+ (2a 2 - i) s (2ai + 1 + i, 2a 3 - i + 1, 2a 2 - i - 1, i - 1 )}. 

Then to = (directly). Rewrite ( c ^) (J) = © jj^ffc = O ^fe^ - The 
proof of the following is in Section 13.31 

Theorem 4. Suppose a\ > and m > 1, then 

" (-2a 3 ) i (-2 a2 ) i _ 2 2m + 3 K a2a3 (2 - 2a 2 ) m _ 1 (2 - 2a 3 ) m _ 1 (k + ffll + l) m 
^ i!(2a 1 +2) i 1 (TO-f)!(-2K-2a 2 -2a 3 + l) 2m 

(ai + §) (4k + 2n + 1) 

x - — — ■ — F (m - 1; k+ 1, a x + 1, a 2 - 1, a 3 - 1) • 

(2ai + 2) m (4K + n) 

Corollary 2. TTie identity is valid and criterion \2.1\) is satisfied for 

even n. 

Proof. The factor (2 — 2a 2 ) _ x (2 — 2a 3 ) m _ 1 vanishes for m > 2a 2 or m > 
2a 3 . Multiply both sides by ( 2ai 1,a3 3+1 ) i the left side becomes 

^ (-2a 2 ) t (-2 fll - 2a 3 - 1) 
U ^!(2a 3 -^)! 

If ai > set m = min (2a 2 , 2a 3 ). The poles of F (as rational function of a 2 ,a 3 ) 
occur in a subset of \ + No- Next we use a weak form of analytic continua- 
tion to apply the Theorem to the case a\ < 0. The terms in the left side with 
2a 3 — i — 1 + (— 2ai — 2a 3 — 1) > vanish, that is, for i < —2a\ — 1. Further 
(2oi + 1)! x (2ai + 2) m = (2ai + m + 1)!). Since 1/ (2ai + m + 1)! is entire for 
fixed m the identity (left side minus right side, for fixed a 2 ,a 3 ,m) can be consid- 
ered as a meromorphic function of a\ vanishing for all a\ > except possibly at 
the poles, which form a subset of h + Z. Now let a\ — > — ^ with 1 < ^ < a 3 . 
The terms in the left side (the sum) vanish for i < 21 — 2 and the right side 
vanishes for 2a\ + m + 1 < —1, that is, m < 21 — 2. By analytic continuation 
^min(2a 2 ,2a 3 ) (201+203+1^202^. = q It rem ains to change the normalizing factor 

in ti to s (1, 2a 3 + 2ai + 1, 2a 2 + 2ai + 1, — 2ai — 1). Note that < c < a+b implies 
2ai + 2a 2 + 1 > 0. The formal expression (put i = —2a\ — 1 = 21 — 1) 

s (1, 2a 3 + 2oi + 1, 2a 2 + 2ai + 1, -2ai - 1) 
s(2ai + 2,2a 3 ,2a 2 ,0) 

= (I ~ K ~ fl 2)^-i (g ~ K ~ fl 3)i-i ( K + "i + 1 ) 2 £-l (I + fl i) 2 ^i 
(I - oa)^! (| - as)^ (i - k - a 2 - a 3 ) 2< ,_ 1 (1 - k - a 2 - as)^ 
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has no poles or zeros at integer values of a±. Thus the identity remains valid when 
multiplied by this ratio. □ 

The formula for the partial sum in the Theorem was discovered by experimen- 
tation, and recognizing that the factorization of the partial sum produces linear 
factors and an irreducible polynomial in (re + 1) (re + n). The validity is proved 
by induction and a contiguity relation for F (that is, to show X^j=i °i = d m for 
sequences {ci} , {d m } it suffices to show do — and c m + d m _i — d m = for m > 1). 
We use a similar approach to the formula 

(2.4) (8re + 2n) £o ((9194 - -7293) (xr (q) , y) 2n ) - £ (As (xt (q) , y) 2n ) = 

for n = 1, 2, 3, . . .. The expression (8re + 2n) £0 ((ft 94 - q 2 q 3 ) K.b («)) "Co (Aj*^ (9) 
with a + b = 2n vanishes when a and 6 are even, so it suffices to take a, 6, c all 
odd. Furthermore assume a > c, otherwise use the symmetry P^ h (94,93,92,(71) = 
P b a ^ b ~ c (9), which produces the same integral because of the invariance properties of 
9194 — 9293 and [i, and replace a, b, c by b, a, a + 6 — c respectively. Note that a + 6 — c 
is odd and c > a implies a + b — c < b. Let a = 2ai + 2a 3 + 1,6 = 2a2 + 1 , c = 2a 3 + 1 , 
so n = at + &2 + &3 + 1 and 01,02,03 > 0. The monomials in P£ b are q a with 
a = (2ai + i, 2a 3 + 1 — i, 2a 2 + 1 — and |a| = 2n. Set 

s(2ai,2a 2 + 2,2a 3 +2,0)ti = 
Co (((8re + 2n) ( 9 l94 - 9293) - D 3 ) q^q^ + ^q^+^ql) 
= (8re + 2n) s (2ai + 1 + i, 2a 3 + 1 - i, 2a 2 + 1 - i, i + 1) 

- (8re + 2n) s (2a x + i, 2a 3 + 2 - i, 2a 2 + 2 - i, i) 
- (2ai + i) s (2ai + i - 1, 2a 3 + 1 - i, 2a 2 + 1 - i, i + 1) 
— « s (2ai + i + 1, 2a 3 + 1 — i, 2a 2 + 1 — i, i — 1) 
+ (2a 3 + 1 - i) s (2ai + i, 2a 3 - i, 2a 2 + 2 - i, i) 
+ (2a 2 + 1 - i) s (2ai + i, 2a 3 + 2 - i, 2a 2 - i, i) ■ 

The proof of the following is in Section 3. 

Theorem 5. Suppose a\ > and m = 0, 1, 2, . . J/ien 

" (-2^-1), (-2 fl3 -l) i f _ 2 2 -+ 3 re (re + ffll ) m+1 (-2a 2 ) m (-2a 3 ) m (at + l) m 
^ z!(2a 1 + l). i 1 m! (-2 K -2a 2 -2a 3 -3) 2m+2 (2a 1 + l) m 

x (4re + 3n + 2) F (to; re + 1, a\, a 2 , o 3 ) . 

Corollary 3. Equation \2.4\ l is valid. 

Proof. Multiply the formula by ( 2 %+ 2 °i +1 ) and set m = min (2a 2 + 1, 2a 3 + 1). 
The factor (— 2a 2 ) m (— 2a 3 ) m vanishes for to > 2a 2 + 1 or to > 2a 3 + 1. □ 

In the paper |4] the proof depended heavily on several integrations by parts. It 
may be possible that such a proof exists in this case (given sufficient ingenuity), but 
the method of integrating P^ b (q) by use of 4i 7 3 -series seemed more straightforward. 
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3. Hypergeometric series tools 

3.1. The single sum. A fundamental transformation for terminating 3F2 
series (for n S No) is 

f-n,a,b \ _ {d-b) n f ~n,c-a,b 

3 2 v c,d 1 ) (d) n 3 ^ 2 Ui+&-rf-« 

By iterating we obtain two useful transformations: 

(3 1) 3 F 2 f ~"' a ' 6 -i^) = ( c + d - a - b )n ^ ( -n,c-a,c-b ^ 
\ c, d 'J (d) n \c,c~a — b + d' 

and (this one provides the sum for the balanced case: — n + a + b + l = c + d) 

(3.2) sF 2 f-^;lW-ir {d ~^f~ h)n 

\ C,d J (c) n Wn 

( —n, a + b — n + 1 — c — d,l — d — n 
x 3F2I ; 1 

\ a — a + 1 — n, — a + 1 — n 

We also need the Whipple transformation (see ^| p. 56]) for balanced terminating 
4F3 series: if n £ No and — n + a + b + c+l = c + d + e then 

(3.3) iF J-n,a,bc, 1 \_(e-a) n (f-a\ 



d,e,f 'J (e)„(/)„ 

/ — n, a, d — b, d — c ^ 
4 3 \c?, e — & — c + d, / — b — c + d' 

With the aid of these formulae we prove the single-sum expression for s (a). To 
concentrate on the intermediate steps we change some variables: 



ui = - - h = - (1 - ai - a 4 ) , 

1 1 , 

u 2 = - - b = - (1 - a 2 - a 3 ) , 

to = ~ (a 3 - a 4 ) , 




Also write a for a 4 . By the parity condition to is an integer; further the sets 
{f'^lT"} an< ^ {n,a — n — are equal. For now we assume to > — n (this is 
necessary because factors (to + l) i with < i < n will appear in denominators). 
The special case to = — n, corresponding to a 3 = 0, 1 will be handled later. Start 
with the double sum from equation ljl.4a|l : note (— a<i) 2i = 2 2% (— t^). ( 1 ~ 2 ° 4 ) i = 
2 2j (— n)j (n — a + |) . and similarly (— a 3 ) 2j - = 2 2j (—to — n)^ (—to + n — a + j).. 
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Further \ — b 3 = i (1 — a 3 — a 4 ) = \ — to — a 

5 = EE 



^ ^ (-n)i (n-a+Q.(-m- n) j (-m + n-a+ \) . (n) i+j 



I= o 3 =o ib'!(«i)iWj(l-m-a) i+j 

(— n) 4 (n - a + |) . / -rn - n, -to + 7i-a+i,K + i \ 

~^ «K u i)i (i - m - a )i 32 \ i-m-a+i,M 2 y 

A (-n) ■ (n - a + |) i (k^ / -to - n, | - to - a - k, i - n N 
•> 1 "77 — wi S — 3i<2 i , ■ , > 1 

* — " il h/i I — — rn — - n \ \ — — rn — n -U i — n — tr _U ? / ^ 



i! (ui) i — m — a)^ I ^ — m — a + i, — n — k, + W2 

n n— i 



I=0 J=0 ilj! (tii)< (-n - k + u 2 ) j (| - m - a) i+j 

= h ■= 



(-« + m 2 - n) n+m 



The transformation (|3.1|) is used on the inner sum. Note that the range of summa- 
tion is now changed to {(i,j) ■ i > 0,j > 0, i + j < ti}, and to can be considered a 
formal parameter. Reversing the order of summation and using equation . 2|> we 
obtain 

gl _ y (' n )j (- m ~ ~ m ~ a ~ "),- / j - n - a + \ 

^ i!(-n-«; + 'U2) J - (| -m-a). \ u x , \ - m - a + j ' y 

" (—«)_,• (—to - n)j. (| - m — a - k),. 

^ j! (-n - « + ii 2 ) 3 - (| -m-a). 

(-m + j - n)^- (1 - to - a - k + 



x(-l) 



("l)n-j (I a + •?')„- 

/ 7 — 71. — 

x 3-F2 



j V2 " 1 J/n-j 

j — 71, —Mi + K + m + li + to + a — 11 



2 

i + K + TO + a — 71, 771+1 / ' 



and reversing the order of summation again we obtain 
{-m-n) n Q-m-o-K) 



5' = (-1) 

n n 

yy 

^ *-~L (| + K + m + a — n) i (m + 1)^ (— n - « + 712) 



(j-m-a) w («i) B 
^ ^ (— n)j {~ u i + k + to + 1), ( ( 5 + m + a — n) . (i - n)- (1 - n - u{)- 

. =0 j ._ Q Kji^TnTiiiTU - V"" T- x H — n-r ^2 / j 

(—n) i (-Tii + K + m+l) i [^+m + a— n) i (ui+u 2 - n — 1) T 



i! (5 + k + to + a - ji) t (m + 1) { (-ri - k + 7i 2 )„_ 

( m + 1 )n(l- m - a - K ) n 



hS", h 



i -m-a) n (u 1 ) n 



At the second last equation the j-sum is done with the Chu-Vandermonde sum 
(2-F1 ( _ c' 6 ' = ^ C (c\ n )" Repl ace i by 71 — i in the sum S"', and use the relations 
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(&)»_, = (b) n I (1 - b - »), and (-,»)„_, / (n - »)! = (-1)" to obtain 

s « = /^r, (-m + k + m + l) w (| + m + a - n) n 
(i + K + m + a- n) (m + 1)„ 

n,tti + U2 — k — 1, 5 — m — a — ac, — m — n \ 
| — m — a, u\ — ac — m — n, — n — ac + «2 ' / 

-n, U\ + 1x2 — ac — 1, n — a + sj, ac \ 
| — m — a, u 2 + to, U\ j 

^ = f^n (~ u i + ac + m + l) w (\-m- a) n (1 - vg - n)„ (1 - m 2 - to - n)„ 
(i - ac - m - a) n (m + 1)„ (— ra - ac + u 2 ) n (— n + iti - ac - m)„ 

The Whipple transformation is valid since the first 4i 7 3-series is balanced (as is the 
second, of course). We combine the factors 

, , , _ (-ac + w 2 - n) n+m (m + l) n (± - to - a - n) n 

J1J2J3 — ~ 




Thus 



(-m + ac + to + l) n (| - m - a) w (m)„ (u 2 + m)» 
(i - ac - to - o) n (to + l) n (-n - AC + U 2 ) n (-Ml + AC + TO + l) n 

( M 2) m 

c (w 2 -Ac) m / -n,Mi + m 2 - ac - l,n - a + i, ac \ 

ft = — p — > — 4^3 i , ;t . 

(u 2 ) m \ ^-m-a,u 2 + m,ui J 

It is possible that > m > —n (that is, 03 < 014) in which case the equation 

(«>» = i-fc = i^E ^ and 80 1 7 u ^ k - (il7- ] uZ m Finally 

for the special case m = —n the double sum S reduces to the single sum 

= K - "On 

while the 4F3 series reduces to a balanced 3F 2 series, so that 

^ f-n,ui +u 2 - AC - l,Av\ = .„ (1 + AC - u 2 ) n (ui - ac)„ 



u 2 -n,ui J {u 2 -n) n (u 1 ) n 

(by use of formula 13. 211 1 and the result multiplied by — }__m — p ro duces 

(l + «-« 2 )_ m 

— r— which equals S. 

This finishes the argument, and now we can show the three-term recurrence ap- 
plies to s (a) . Returning to the original variables, we note bo—m = ~ (a 2 + 03 — a.3 + 0*4,) 
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b 2 , thus 

(ug ~ «) m _ (| + &o + « - m) m (n+r) b2 (l) b2 



M m ~ (i + &o-m) m (i) fe9 ( K +|) 



m V 2 / £> 2 2 / b 2 

2")b (2)^2 



vDbo ( K + 2)b 2 



This proves formula (|1.7|l . 

3.2. Three term recurrence and symmetries. 

Proposition 4. for neN 

n (n + 2vi) yi+ k + vi - + i + u 2 + 1*3 - raj -F (ft - 1; «, «i, ^2, ^3) 

+ jn (n + 2«i) - t» 2 - v 3 - ~J + + ^ + ui j (n - 2u 2 ) (n - 2v 3 ) 
x [n + t>i - i ) F (n;n,v 1 ,v 2 ,v 3 ) 



2 
1 

2/ V" ' ~" ' 2 



n + Vi — - ) ( n + V\ + - ) (n — 2u 2 ) (n - 2v 3 ) F (n + 1; k, vt, v 2 , v 3 ) 



Proof. Expand the left side as a series in gi := (/«). (— k — — v 2 — 1*3) • for 
< i < L 2 ^] ■ Start with the equation 



1\ ( 1 



n + K + V! - -J yn + -+v 2 +v 3 -n 
^ - vi - n + i \ (n - v 2 - v 3 - ^ + ij -(« + «) (-« - «i - v 2 - i> 3 + i) . 

The outline of the calculation is this: write F (n — 1) = ^2 i>0 oagi and F (n) — 
J2i>o a i9i (suppressing the other arguments); note that gi (k + i) (— k — v\ — v 2 — v 3 + i) 
gi+i then collect term-by-term in 



(n + 2vi) {n Q - vi - n + ij (n - v 2 - v 3 - ^ + ij ^ a^i 

^ ' ^ ' i>0 



»>1 x 7 v 7 i>0 

- ( " - - ''i j ! " '' ~ ^ ) ( n ~ 2v 2) (n - 2v 3 ) ^ a 'i9i 

i>0 
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(_») (-A±i) 

Write this sum as th — 2 y J n — 2 \ U TCiQi, then (in corresponding or- 

der) 

n + 2v 1 ( 1\ , , . , / 1 

c 4 = — - I n + ui - - J {- (n - 2z) (n + 1 - 2«J I n - v 2 - v 3 - - + i 

+ n (n + 1 - 2i) - v 2 - v 3 - + i (2v 2 + l-2i) {2v 3 + 1 - i)} 

n + l-2i ( l\ . „ w 

H — — n. + «i - - [n - 2v 2 ) In - 2v 3 ) 

n+1 \ 2 J 

(n 2v 2 ) (n - 2v 3 ) ^ + ^ _ 1 \ f . + + / + ^ + 1\ + J _ 



n + 1 V 2 /l V 2 

= (n - 2w 2 ) (n - 2v 3 ) (n + v x - + «i + ^ + i 

Finally (n + Ui + ^ + i) / (| — n — «i) = (n + t>i + |) / (— ^ — n — t>x) which 
proves the identity. □ 

Corollary 4. 77ie recurrence \1.5)) for s' (a) is valid. 

3 U)„ 

Proof. Suppose cti = a 2 = 03 = «4 mod 2 then set c' (a) — Y[i=i 7 — it - 

where &i = 5 (ct!i + 014) , 1 < i < 3. Then s' (a) /c' (a) satisfies the recurrence for F 
in the Proposition, with n = 014, v\ = 5 (ai — 014) ,v 2 = b 2 ,v 3 — b 3 . Multiply the 
recurrence by c' (a) then the coefficient of F (n — 1 ) is multiplied by 

c' (a) ai + 0:4 — 1 



C' (ai — 1, OL 2 + 1, «3 + 1, CK4 — 1) 2k + a\ + Qf4 — 1 

and the coefficient of F (n + 1 ) is multiplied by 

c' (a) 2k + ai + 0:4 + 1 

c' (ai + 1, a<2 — f , a 3 - 1, a 4 + 1) ai + a 4 + f 

Divide out the common factor h (ai + 0:4 — 1) = (n + v\ — ^) to obtain (|I.5|I . □ 

There is another, perhaps unexpected, symmetry: 

Proposition 5. s' is completely symmetric in its arguments. 

Proof. Already formula I f . 71 shows the symmetry in (ai, a 2 , a 3 ). One can ar- 
gue from the 012 <-> 013 and oc\ <-> 04 invariance together with the a «-> (a 2 , ai, 014, a 3 ) 
invariance of s. The Whipple transformation (|3.3|) gives a direct proof. It suf- 
fices to show s' {a\, 0C2, as, a.4) = s' (a4, a 2 , ot 3 , a{). If all the a^'s are even take 
n = ^ ,a = K,d = ^ — 61, then 



s' (ai,a 2 , a 3 ,a 4 ) 



(2)03/2 (2)03/2 



( K + 5)a 2 /2 ( K+ \)a 3 /2 

5 - 61, k + i (1 + a 3 ) , k + \ (1 + a 2 ) 



/ -^,K,^fi, K +i(l + a 2 +a 3 ) / 

4^3 In. 1 - - 1 - - . ; 1 



TYPE B 2 INTERTWINING 



21 



and this expression is symmetric in (ai, 0:4). A similar argument works when is 
odd: 



s' (or, a 2 , a 3 , a 4 ) = -. 1 



(2)(a 2 +l)/2 \V(a a +l)/2 



\ K + 2j( Q2 + l)/2 ( K + 2)( Q3 + l)/2 

/i^, K ,i^, K +i(l + a 2 + a 3 ). 1 N 



□ 

3.3. Contiguity Relations. We start by writing the expressions in Theorems 
0] and El in terms of the function F (see definition^. Suppose ot\ = a 2 = a 3 = 
«4 mod 2 then set 

where hi ~ \ {on + a 4 ) , 1 < i < 3; thus s (a) = c (a) F (a 4 ; k, \ {a\ — a 4 ) , 6 2 , 63) . 
For Theorem 0] we set i = m > 1 and extract the common factor 

c (2ai + to, 2a 3 - m, 2a 3 - to, m) (-2a 3 ) m (-2a 2 ) m 

O17 



s (2oi + 2, 2a 3 , 2a 2 , 0) m! (2a x + 2) m (4k + n) (2k + n) 
= 2m _! (4« + 2n + 1) (k + ai + !),„_! (01 + |) m _ 1 (-2a 3 ) m H^L 
(4k + n) (1 - 2k - 2a 2 - 2a 3 ) 2m m! (2o x + 2) m 

Recall n = 2j=i a »- When i — we have 

t = 2 (4k + 2n + 1) - 2 ^ + *> ± fll > C ^ 2 ° 3 ' 2 ° 3 ' ^ 
V 7 (2K + n)s(2ai + 2,2a 3 ,2a 2 ,0) 

= 0. 

The plan is to use induction by showing - mfyai+ty 2 ^™ ^ m dm—l ~ d m = 0, where 
d m denotes the claimed expression for the sum. The following multipliers are needed 
in the calculation of t m : 

c (2ai + m + 2, 2a 3 — m, 2a 2 — to, to) (k + ai + to) (2oi + 2m + 1) 
c (2ai + to, 2o 3 — m, 2o 2 — to, to) (4k + 2n + 1) (2k + n) 

c (2oi + to + 2, 2a 3 - to, 2a 2 - to, to - 2) _ (2k + 2a 2 - 1) (2k + 2a 3 - 1) 
c (2ai + to, 2a 3 — to, 2a 2 — to, to) (2a 2 — 1) (2a 3 — 1) 

c (2ai + to + 1, 2a 3 — to — 1, 2a 2 — to + 1, to — 1) 2k + 2a 3 — 1 
c (2ai + to, 2a 3 — to, 2a 2 — to, to) 2a 3 — 1 ' 

c (2ai + to + 1, 2a 3 — to + 1, 2a 2 — to — 1, to — 1) 2k + 2a 2 — 1 
c (2ai + to, 2a 3 — to, 2o 2 — to, to) 2a 2 — 1 

Write d m = A m F (to — 1; k + 1, a± + 1, a 2 — 1, a 3 — 1) then 

A m . (2a 2 - m) (2a 3 - to) 

— = 4TOK(K + ai+ TO) (2fl2 _ 1} (2fls _ 1} , 

A m _i 2 (to — 1) (2ai + to + 1) (k + a 2 + a 3 - to) (k + a 2 + a 3 + i - to) 
yl m (k + ai + to) (2aj + 2m — 1) (2a 2 — to) (2a 3 — to) 
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The expression m^ai+l) 2 ^" 1 ^™ + ^m-i _ ^m) /Cm with each a, is replaced by 
(formal parameters instead of integers) becomes the left side of the equation in 
the following. 

Theorem 6. Suppose v\,v 2 ,v 3 ^ —5 + No, (v = v\ + v 2 + v 3 ) and m = 
1, 2,3,.. .then 

2 (2t>i + 2to + 1) (k + vi + to) (4k + v ) F (to; k, «i + 1, v 2 , v 3 ) 

- {2 (k + ui + to) (4k + w ) + to (3k + v a )} (2v\ + to + 1) F (to; k, ui, u 2 > u 3 ) 

/ ,n (2^ + 2k-1)(2i; 3 + 2k-1) 

- to (m - 1) ^ — _ ^ ^— _ '- (3k + u ) F(m-2; k, ^ + 2, v 2 - 1, w 3 - 1) 

- TOK— — — ^— — — — — — (2w 3 — m) F (to — 1; K,Vi + l,v 2 ,v 3 - 1) 

2w 3 - 1 

- ^ t 2 " " 1 " 2 " 2 - 1 ) (2w 2 — to) F (to — 1; k, «i + l,t) 2 - 1, w 3 ) 

2i> 2 — 1 

8k(2ui+to+1) , . / 1 

+ m (m 1} (2, 2 - 1) (2,3 - 1) (2v, +2 m -l) {K + V2+V3 - m) {K + Vi+Vz+i- 
x F(m - 2;/t+ + l,v 2 - l,v 3 - 1) 

(2v 2 - to) (2w 3 - to) . , ^ . , , 1 . 

- 4tok— — — — - (k + vi + to) F (to - 1; k + 1, Vi + 1, v 2 - 1, t> 3 - 1) 

(2w 2 - 1) (2w 3 - 1) 

= 0. 

Proof. We proceed by considering the left side as a polynomial in k of the 
form 

^2 ( c i + d i K ) -V1-V2- v 3 ) i . 

The coefficients are rational functions of Vi,v 2 ,v 3 ,m. Write the identity in abbre- 
viated form as Y^ n =i VnF (in) = using the same order as above (for example, 
72 = {m; k, vi, v 2 , v 3 ) and fi 5 = — mn^^pp^ (2w 2 — to)). Denote term #z of 
F (in) by U (7„), (where t (i n ) = 1 for all n) so that 

*™ (1-^-^(1-^,(1-^ <! ' 
for example; with the understanding that F (to) = for to < 0. The verification is 
carried out by symbolic computation. The underlying idea is to set up the equations 
as rational functions of all of the variables. For example, (a) i is not rational in i, 
but for a fixed integer k the expression (a) i / (a + k) i is rational in a and i because 
of the identity (a) i / (a + k) i = (a) k / (a + i) k . By use of symbolic calculation we 
evaluate 

/ \ (7l) . . U (in) 

^ 2 > n~3 tl ^ 2 > 

7 



Po («) = Mi (1 + *i (7i)) + M2 + 



n=3 



for 1 < z < [^J . Indeed (k) is a polynomial in k of degree 4 (note that jq^j is 
not polynomial in k for i > 1, and the factor k in ^ 6 and ^7 is necessary since 
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has k in the denominator for n = 6, 7), further pi (k) is rational in all the variables 
(ui, V2, V3, K, i, to). Although i is an indexing variable the difference between the 
respective parameters in {7^} is one of 0, ±1, ±2 and thus ^fej is rational in i for 

each n. By computer algebra we find the coefficients {ci,fc}^ = Q, which are functions 
of i, to, v%, V2, U3, so that 

Pi (k) = c lfi + Ci.iK + (ci, 2 + Cj,3«) (re) + c ii4 5i !2 (ft) , 

where g ilfe (re) := (re + i) k (-re - v + for fc > 0, thus g ,i (ft) Si.fc (ft) = 5o,i+fc (ft)- 
For example 

_ 4 (2i - to) (2i + 1 - to) 
Ci ' 4 ~ (t + 1) (5 - «2 + ») (5 - «a + ») 
for < i < , but the other coefficients are more complicated. Because the 
degree of re in [i\ is 2 the value of co,4 agrees with the generic Cj ; 4 with i = 0. For 
i > let 

_ (72) 
ffo.i (ft) 

/ m\ / 1— m \ 

so that is independent of re (thus = — T , \- J T\ — - — h~ 71 s - )- The 

*!( 3 -«i -to). (2 -^(5 -vs), 

left side of the identity equals 

Lm/2j 

X] P* ( K )9o,i ( K )n 

i=0 
Lm/2j 

= ^ Tj ((Ci )0 + Ci,i«) 5o : i + (Ci,2 + Cj,3ft) 50 : i+l (ft) + Ci,4ff0,i+2 (ft)) 
i=0 
\_m/2\+2 

= X! .90,i ( r i ( c i,0 + c i4 K ) + r »-l ( C i-1,2 + ^-1,3^) + ^^2^-2,4) • 
i=0 

By symbolic computation 

(Cj,0 + Cj lK) + (Cj_l 2 + Ci_l,3«) + ^-^-Ci-2.4 = 

for 2 < i < [f J ■ Also 

C0,0 + Co,lK = 0, 

ri (ci, + ci,ik) + r (c ,2 + Co,3ft) = 0. 

The special cases at the top end of summation occur at i — y or i = (for to 
being even or odd, respectively) and in fact Ci.4 = and (0^2 + Cj l3 re) + -^Ci-1.4 = 
for these values of i. □ 

This completes the proof of Theorem 0] 

We use the same approach to Theorem 03 where we set i = to > and extract 
the common factor 

c (2ai + to - 1, 2a 3 - to + 1, 2a 3 - to + 1, to + 1) (-2a 3 - l) m (-2a 2 - 1)_ 



C 
= 2 



s (2ai, 2a 3 + 2, 2a 2 + 2, 0) to! (2oi + l) m (4re + 2n + 1) (2k + n) 

■ M ( K + °l)m + |L (~ 2fl 3 - !)m (~ 2fl 2 - l) m 

(-3-2 K -2a 2 -2a 3 ) 2m+2 m!(2a 1 + l) m 
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Next calculate (recall n = ai + a 2 + a 3 + 1) 

c (2ai + m + 1, 2a 3 + 1 — m, 2a 2 + 1 — to, to + 1) (k + ai + m) (2ai + 2m + 1) 
c(2ai + to - 1, 2a 3 + 1 - to, 2a 2 + 1 - to, to + 1) ~ (4k + 2n + 1) (2k + n) 

c (2ax + to, 2a 3 + 2 — to, 2a 2 + 2 — to, to) 
c (2ai + to — 1, 2a3 + 1 — to, 2a 2 + 1 — to, to + 1) 

(k + a 2 + »3 + 1 - m) (« + a 2 + a 3 + | - to) 
(4K + 2n+l)(2K + n) 
c (2ai + to + 1, 2a 3 + 1 - to, 2a 2 + 1 - to, to - 1) _ (2k + 2a 2 + 1) (2k + 2a 3 + 1) 
c(2ai + to - 1, 2a 3 + 1 - to, 2a 2 + 1 - to, to + 1) ~ (2a 2 + 1) (2a 3 + 1) 

c (2ai + to, 2a 3 — to, 2a 2 + 2 — to, to) 2k + 2a 3 + 1 

c (2ai + to — 1, 2a 3 + 1 — to, 2a 2 + 1 — to, to + 1) 2a 3 + 1 ' 
c (2ai + to, 2a 3 + 2 — to, 2a 2 — to, to) 2k + 2a 2 + 1 

c (2ai + to — 1, 2a 3 + 1 — to, 2a 2 + 1 — to, to + 1) 2a 2 + 1 

Write the right side of the summation formula in Theorem[3]as d m = A m F (to; k + 1, a 1; a 2 , a 3 ). 
Then 

A m . , , . w, , q , (2a 2 - to+ 1) (2a 3 - to + 1) 

— = 4k (k + ai + to) (4k + 3n + 2) — — — — — , 

C m (2a 2 + 1) (2a 3 + 1) 

A m -i 2to (2ai + to) (k + a 2 + a 3 + 1 — to) (k + a 2 + a 3 + I — to) 

^4 m (k + ai + to) (2oi + 2to - 1) (2a 2 - to + 1) (2a 3 - to + 1) 

The expression f ^"^i^a^ 2 " 3 ^"' *m + rf m-i - d TO J /C m with a 4 replaced by v t 
(formal parameters instead of integers) for 1 < i < 3 becomes the left side of the 
equation in the following. 

Theorem 7. Suppose v 1 , v 2 , v 3 g -| +N 0; ft>o = «i + u 2 + V3 + 1) and m G N 

2 (2« x + 1 + 2m) (k + + to) (4k + v Q ) F (m + 1; k, «i, w 2 + 1, w 3 + 1) 

— 4 (4k + v ) ^k + v 2 + v 3 + ^ - to^ (k + w 2 + v 3 + 1 — to) F (to; k, ui, u 2 + 1, v 3 + 1) 

- (2vi + to) (4k + 2v + 1) (2k + v ) F (m+ 1; k, u x - 1, i> 2 + 1, w 3 + 1) 
(2k + 2w 2 + 1)(2k + 2w 3 + 1) 



(2u 2 + 1) (2v 3 + 1) 
(2k + 2w 3 + 1) 



, (4k + 2v Q + 1) (2k + v ) F (m - 1; k, Uj. + 1, v 2 , V 3 ) 



(2v 3 + 1) 
(2k + 2w 2 + 1) 
(2v 2 + 1) 



(2w 3 + 1 - to) (4k + 2v + 1) (2k + v ) F (to; k, v t ,v 2 + 1, U3) 
(2w 2 + 1 - to) (4k + 2v + 1) (2k + w ) F (to; k, u x , v 2 ,v 3 + 1) 



8tok (2«i + to) / 3 , 

k + v 2 + v 3 + -— m) (k + u 2 + v 3 + 1 - to) 



(2w 2 + 1) (2w 3 + 1) (2vi + 2m - 1) 
x (4k + 3u + 2)F(m - 1;k+ 1, «i, w 2 , w 3 ) 
4k (2w 2 + 1 - to) (2u 3 + 1 - to) 



(2w 2 + 1) (2u 3 + 1) 

0. 



(4k + 3i> + 2) (k + vi + to) F (to; k + 1, Vi, u 2 , v 3 ) 
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Proof. The method is similar to the previous one; here we use 73 as point of 
reference for the ratio calculations. Write the identity as Y^ n =i Mn-F (in) — and 
use the notation fj (7„) as before. As before g, L ^ (k) '■= {k + i) k (— k — vq + i) k (but 
here vq = Y^=i v j + •"■) different from the previous theorem). Set 

/ \ U+i (7™) . v i; • 1 

2 8 

po («) = E ( 7 «) + E 

n— 1 n— 1 

Then (k) is a polynomial of degree 5 in k. By computer algebra we find the 
coefficients {cjfc} fe=0 , which are functions of i, m, Vx, V2, V3, so that 

Pi («) = Q.O + Cj.lK + (c h2 + Ci <3 K) gis (k) + (Cj,4 + CifiK) gt,2 («) • 

Then the left side of the identity equals 

L(m+i)/2j 

E Pt( K )9o,i(K)n 

z=0 

L(m+1)/2J 

= E ^ (( Cl >° + Ci ' lK ) ff0 ^ + ( Ci > 2 + Cl - 3 ^ S'O.i+l ( K ) + (Ct,4 + Ci,5«) 50,1+2 («)) 
i=0 
L(m+l)/2j+2 

= E ff0 '» ^ ( Ci >° + Ci > lK ) + ri - 1 (^-i. 2 + C i-l,3 K ) + r i-2 (Ci-2,4 + Ci-2,5^)) , 
t=0 

(73) 

where r$ := — - — ^— . The degree of k in ^1 and /i2 is 3 and thus cq 4 + cq 5K agrees 
90,i («) 

with the generic value of Q.4 + c^k at i = 0. In the range 2 < i < L 22 ^ 1 -] the 
value of (c tfi + Cj,i«) + (cf-1,2 + Cj-i^w) + (c 4 _2,4 + is found to 

be zero by symbolic computation. Also cq^q + cq^k — and ri (ci o + ci ik) + 
r o (co.2 + C0.3K) = 0. The special cases at the top end of summation occur at i = y 
or i = "j" 1 (for m being even or odd, respectively) and in fact c^4 + = and 
(ci,2 + Cj ; 3/t) + (q_i,4 + Ci-i^rt) = for these values of i. □ 

This completes the proof of Theorem 

4. Closing Comments 

Firstly we remark on the difficulty and complexity of the proof of the inter- 
twining property. We speculate that one reason is that the measure on the linear 
transformations r (q) is not uniquely defined. As was seen in the presentation it is 
only the integrals of P£ b (q) that matter. By Rosler's result [5] for each x G M 2 
and k > there is a positive Baire measure fi x such that Vf (x) — J fdfi x and 
the support of /z is contained in the closed convex hull of the i?2-orbit of x. The 
explicit measure in Theorem |3 requires k > § and is not positive (combine the two 
parts into one and evaluate the kernel at u = e, <f>i = 4>2 = §> ipx = £, "02 = 7r — e for 
small positive e; the value is 2 (2k - 3) e 2 cos + e 4 (l + (7 - ^) cos 0) + O (e 5 ) ; let 
j < 8 < 7r). However the integral formula for the Bessel function K° (x, y) is posi- 
tive and works for k > \ . It would be very interesting if one could develop explicit 
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results for the Coxeter groups A n or B n for n > 3. The corresponding compact Lie 
groups (U (n + 1) and Sp (n), repectively) are fairly concrete but clearly we need 
more powerful techniques than we used here for n = 2. 

Secondly there is the two-parameter problem. The group B2 allows two param- 
eters in the associated differential-difference operators (see Hl-lj l'). The approach to 
finding the measure [i by starting with the Lie group Sp(2) did not (as yet) help 
us in the two-parameter problem. This may be the hidden reason why the part of 
the intertwining formula associated with the representations of types 1 and 2 (re- 
alized on X1X2 and x\ — x\ respectively) is so complicated (involving derivatives); 
we were somehow (heuristically) close to the kernels for k\ = n + 1 , «2 = k and 
K\ = K, K2 = k + 1. 

Finally we comment on the role of computer algebra. We produced K n (x, y) 
for n < 6 by solving the equations TfK n (x, y) — j/ji^„_i (x, y) , i = 1, 2. Then by 
use of the symmetrized kernels | J2 w eB 2 ^ n ( xw i V) we conjectured that the mea- 
sure /j, is the right one. Borrowing the polynomial 2 (qi + (74) + (71(74 — (7293 from the 
S3 paper |3| quickly helped to expand the conjecture. The part of the intertwining 
operator dealing with types 1 and 2 (see Section 12. 1|) turned out to be puzzling. 
Eventually we had a formula for V that worked for each K n (x, y) tried (up to 
n = 12), but no proof. Then experimenting with P^bil) ^ eo - *° discovering the 
single-sum for s (a) (see Section rO|l . The proof itself depends on classical trans- 
formations of hypergeometric series. In turn the single sum expression (which had 
three free parameters) made feasible enough experiments to formulate the identity 
in Theorem 0] The term- by-term calculations which prove the contiguity relations 
in Section \'A . 31 would be very tedious without symbolic computation assistance, and 
much space would be needed to write down every intermediate step. However it is 
straightforward to verify these relations for any particular m (magnitude depending 
on the size of the computer; to = 1 ... 6 is not too big). There is some satisfaction 
in coaxing a symbolic computation system actually to prove a conjecture after it 
helped in the conjecture's formulation. 
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